Open Access 



Research 



Differences between self-reported 
and verified adverse cardiovascular 
events in a randomised clinical trial 




Mark J Bolland, Alan Barber, Robert N Doughty, Andrew Grey, Greg Gamble, 
Ian R Reid 



To cite: Bolland MJ, 
Barber A, Doughty RN, etal. 
Differences between self- 
reported and verified adverse 
cardiovascular events in a 
randomised clinical trial. 
BMJOpen 201 3;3:e002334. 
doi:1 0.1 1 36/bmjopen-201 2- 
002334 

► Prepublication history for 
this paper are available 
online. To view these files 
please visit the journal online 
(http://dx.doi.org/10.1136/ 
bmjopen-201 2-002334). 



Received 12 November 2012 
Revised 20 February 2013 
Accepted 25 February 2013 



This final article is available 
for use under the terms of 
the Creative Commons 
Attribution Non-Commercial 
2.0 Licence; see 
http://bmjopen.bmj.com 



Department of Medicine, 
University of Auckland, 
Auckland, New Zealand 



Correspondence to 

Dr Mark J Bolland; 
m.bolland@auckland.ac.nz 



ABSTRACT 

Objectives: In clinical trials, adverse events are 
usually self-reported but may be adjudicated if serious 
or of particular interest. After adjudicating 
cardiovascular events for a 5-year calcium supplement 
trial, we observed discrepancies between self-reported 
and verified events. We systematically analysed those 
differences to assess their importance. 
Design: Secondary analysis of adverse 
cardiovascular events in a 5-year, randomised, 
placebo-controlled trial of calcium supplementation 
(1 g calcium daily) in 1471 postmenopausal women 
(mean age 74 years). 
Setting: Clinical research centre. 
Methods: The participant's medical records were 
reviewed for all self-reported myocardial infarctions 
(Mis) or strokes, and the event independently 
adjudicated. Cause of death was obtained from 
hospital records or death certificates. To identify 
unreported events, the national hospital discharge 
database was searched and related hospital records 
were reviewed. 

Results: 45 women reported 64 Mis, of which 33 
(52%) were verified after adjudication. An additional 
25 Mis were identified: 1 during adjudication of other 
events, 21 from the hospital discharge database, 3 
from death certificates. 68 women reported 86 strokes 
of which 50 (58%) were verified. An additional 13 
strokes were identified: 7 during adjudication of 
reported transient ischaemic attacks, 5 from the 
hospital discharge database, 1 from death certificates. 
Therefore, 43% of verified Mis and 21% of verified 
strokes were not reported to investigators. For non- 
adjudicated discharge codes, 10% of Mis and 22% of 
strokes were not verified after adjudication. Nineteen 
per cent of verified Mis and 27% of verified strokes 
were not identified in discharge coding or death 
certificates. Neither the event source nor the level of 
adjudication altered the relationship between 
treatment allocation and cardiovascular events. 
Conclusions: When adverse event accuracy is 
critical, researchers should consider adjudicating self- 
reported events and hospital discharge codes, and 
attempt to identify unreported events. 
Trial registration: Australia New Zealand Clinical 
Trials registry: ACTRN 012605000242628. 



ARTICLE SUMMARY 



Article focus 

■ The differences between self-reported and verified 
adverse cardiovascular events in a clinical trial. 

Key messages 

■ Substantial proportions of self-reported myocar- 
dial infarctions (Mis) and strokes were not able 
to be verified, and substantial proportions of 
verified Mis and strokes were not reported by 
participants or were not identified in discharge 
coding or death certificates. 

■ When adverse event accuracy is critical to a clinical 
trial, unadjudicated self-reports or hospital dis 
charge codes cannot be relied upon. Consideration 
should also be given to identifying unreported 
events. 

Strengths and limitations of this study 

■ A rigorous search for adverse events occurred 
and all events were independently adjudicated. 

■ Cardiovascular events were secondary endpoints 
and participants may have placed less value on 
reporting these events than the primary endpoints. 



INTRODUCTION 

In clinical research studies, participant self- 
report of adverse effects and outcome events 
are essential in determining the efficacy and 
tolerability of the intervention being studied. 
Participant self-reports may be accepted as 
accurate or may lead to independent adjudi- 
cation, depending on the relevance to the 
study and the overall size and complexity of 
the study In large clinical trials, adverse 
events are usually self-reported and not inde- 
pendently verified unless considered serious 
or of particular interest. However, the few 
studies that have specifically addressed the 
accuracy of self-reported medical events 
suggest relatively poor agreement between 
self-reports and medical records. 1 2 

Previously, we completed a 5-year trial of the 
effects of calcium supplementation in healthy 
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postmenopausal women in which we observed an unex- 
pected increase in the rate of vascular events in women 
allocated calcium. 3 All vascular events were initially self- 
reported, and then adjudicated by blinded study investiga- 
tors. We also conducted a systematic search for valid events 
that were not reported by participants. We present the 
results of an analysis of the relationship between self- 
reported and adjudicated events, including unreported 
events. The current analyses were not planned in the ori- 
ginal trial protocol or the subsequent protocol for the 
adjudication of vascular events. Ethical approval for the 
current analyses was not required. 



METHODS 
Study design 

The Auckland Calcium study was a 5-year, randomised, 
placebo-controlled trial of calcium supplementation 
in 1471 normal postmenopausal women (mean age 
74 years) , designed to assess the effects of 1 g daily 
calcium on fracture incidence. 

Recruitment started in 1998 and was completed in 
2006. Cardiovascular outcomes were a prespecified sec- 
ondary endpoint. The methods have been described in 
full previously. 4 In brief, women were recruited by adver- 
tisement and from mail-outs using electoral rolls, and 
were 5 years or more postmenopausal and aged 55 years 
or older. Women were ineligible if they were receiving 
therapy for osteoporosis, had other major ongoing 
disease, or had serum 25-hydroxyvitamin D<25 nmol/1. 
The study received approval from the regional Ethics 
Committee and the trial was registered with the 
Australia New Zealand Clinical Trials registry: ACTRN 
012605000242628. 

Cardiovascular event assessment 

Participants were reviewed every 6 months. Adverse 
events were recorded at each visit but questions about 
specific symptoms or illnesses were not asked. A pre- 
planned secondary analysis was a comparison between 
the groups in the frequencies of myocardial infarction 
(MI), stroke, transient ischaemic attack (TIA) and other 
cardiovascular events. 3 During 2006-2007, cardiovascular 
events were adjudicated. The participant's medical 
records were reviewed when an MI, stroke or TIA was 
self-reported (or by family members for fatal events). 
For participants who died during the study, the cause of 
death was obtained from hospital records or the death 
certificate. Data for each event were compiled by a phys- 
ician and then adjudicated by a cardiologist (MI) or 
neurologist (stroke or TIA). All were blinded to the 
treatment group of each participant. MI was defined in 
accordance with the Joint European Society of 
Cardiology/ American College of Cardiology Committee 
criteria for acute, evolving or recent MI 5 and stroke and 
TIA were defined in accordance with the WHO 
definition. 6 



Discharge codes 

To identify events that were unreported, we searched the 
national database of hospital discharges for cardiovascu- 
lar events (ICD-9 discharge codes 410 for MI and 430, 
431, 433, 434 for stroke) that occurred during the study 
using each participant's unique National Health Index 
identifier. The hospital records related to these admis- 
sions were reviewed and adjudicated in the same 
manner as for self-reported events. We added all unre- 
ported, adjudicated hospital discharge events to the 
adjudicated self-reports to obtain a complete set of veri- 
fied events. We compared this complete set of verified 
events with the events obtained solely from discharge 
ICD-9 codes or death certificates. 

Statistics 

Agreement between reported and verified events was 
assessed using the k coefficient. The number of women 
experiencing an incident event in each treatment group 
was compared using Fisher's exact test. Differences 
between baseline characteristics were compared using t 
tests for continuous variables, and Fisher's exact test for 
categorical data. All statistical analyses were performed 
using the SAS software package (SAS Institute, Cary, 
North Carolina, USAV.9.1). p<0.05 was considered statis- 
tically significant. 

RESULTS 

Myocardial infarction 

Table 1 shows the number of self-reported, verified self- 
reported, verified unreported and total verified events. 
Thirty-three of 64 (52%) self-reported Mis were verified. 
Table 2 shows the final diagnosis from the medical 
records for the 31 Mis that were not verified. In 10 
instances, there was no report of the event in the 
medical records, but in each case a verified MI had pre- 
viously occurred during the study. For 9 of the remain- 
ing 21 events (43%), MI was specifically excluded with 
final diagnoses made of angina, unstable angina, or MI 
excluded. Twenty-five unreported events were identified: 
1 during adjudication of other events, 21 from the 
national hospital discharge database search and 3 from 
death certificates. Thus, of 58 verified Mis in 52 women, 
only 33 (57%) were reported to study investigators and 



Table 1 Numbers of events by source and adjudication 


status 




Myocardial 




Source/adjudication infarction 


Stroke 


Total self-reported events 64 (45) 


86 (68) 


Verified self-reported 33 (30) 


50 (47) 


events 




Verified unreported events 25 (24) 


13 (12) 


Total verified events 58 (52) 


63 (59) 


Data are number of events (number of women). 
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Table 2 Final diagnosis from medical records for self-reported, non-verified myocardial infarctions (n=31) 


and strokes (n=36) 


Myocardial infarction 




Stroke 




Final diagnosis 


N 


Final diagnosis 


N 


Unstable angina 


c 
O 


Transient ischaemic attack 


6 


Congestive heart failure 


5 


i— 1 1 
hall 


3 


Angina 


2 


Acute confusional state 


3 


Sudden unexplained death 


o 

tL 


Sudden unexplained death 


2 


Atrial fibrillation 


1 


Benign positional vertigo 


2 


Palpitations 


1 


Dementia 


2 


Postural hypotension 


] 


Postural hypotension 


1 


riclld.1 IdllUic 




1 InctooHx/ c\y\ foot 
UllblcclUy UN Iccl 


1 


Shortness of breath cause unknown 




Seizure 


1 


Myocardial infarction excluded 


2 


Head injury 




No details of event in medical record 


10 


Blackout 








Vasovagal event 








Ischaemic retinal vein occlusion 








Stroke excluded 





No details of event in medical record 10 



25 (43%) were identified from other sources. The k 
value for agreement between reported and verified Mis 
was 0.63 (95% CI 0.51 to 0.74). 

When only non-adjudicated events identified from the 
discharge database search were considered, there were 
48 Mis in 41 women. When death certificate data were 
added, there were 52 Mis in 45 women. Five (10%) of 
the 48 Mis identified from the discharge database 
search were not verified. Another 11 verified Mis (19%) 
were identified from self-reports but not from hospital 
discharges or death certificates. 

Stroke 

Table 1 shows that 50 of 86 (58%) self-reported strokes 
were verified. Table 2 shows the final diagnosis for the 
36 strokes that were not verified. In 10 instances, there 
was no report of the event in the medical records, but in 
each case a verified stroke had previously occurred 
during the study. For 8 of the remaining 26 events 
(30%), stroke was specifically considered and excluded 
with final diagnoses made of TIA, or stroke excluded. 
Thirteen unreported events were identified: 7 during 
the adjudication of reported TIA, 5 from the discharge 
database search and 1 from death certificates. Thus, of 
63 verified strokes in 59 women, 50 (79%) were reported 
to study investigators as stroke, 7 (11%) as TIA and 10% 
were identified from other sources. The k value for 
agreement between reported and verified stroke was 
0.73 (95% CI 0.64 to 0.82). 

When only non-adjudicated events identified from the 
discharge database search were considered, there were 
50 strokes in 46 women. When death certificate data 
were added, there were 57 strokes in 48 women. Eleven 
of the 50 strokes (22%) identified from the discharge 
database search were not verified. Another 17 verified 
strokes (27%) were identified from self-reports but not 
from hospital discharges or death certificates. 



The relationship between treatment allocation and MI 
or stroke for the different event sources and levels of 
event adjudication is shown in table 3. There were con- 
sistently increased relative risks for MI and stroke with 
calcium supplements, although the relative risks 
changed in some cases depending upon event source or 
level of adjudication, and the different numbers of 
events in each comparison led to more marked changes 
in p values. 

Finally, we assessed possible relationships between 
baseline characteristics and discrepancies between self- 
reported and final verified events. Women in whom 
there were discrepancies were older (p<0.001), and were 
more likely to have reported at the baseline study visit 
having had a previous MI (p=0.013) or previous TIA or 
stroke (p=0.035) than women without such discrepan- 
cies. However, there were no differences in physical activ- 
ity (p=0.7) or other comorbidities between the groups 
(p>0.1 for hypertension, dyslipidaemia, diabetes). 

DISCUSSION 

We observed differences between self-reported MI or 
stroke and the final verified diagnosis obtained from the 
medical records or death certificates. Almost half of self- 
reported Mis (48%) and strokes (42%) were not able to 
be verified, and 43% of verified Mis and 10% of verified 
strokes were unreported, with a further 11% of verified 
strokes reported as TIA. Differences were also observed 
between events obtained solely from hospital discharge 
codes and final verified events. Ten per cent of Mis and 
22% of strokes obtained from discharge codes were not 
verified, and 19% of verified Mis and 27% of verified 
strokes were identified from self-reports but not from 
discharge codes or death certificates. Relying solely on 
self-reports of MI or stroke, or solely on discharge codes 
in our clinical trial, would have led to significant 
numbers of participants being misclassified with regard 
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Table 3 Relationship between treatment allocation to calcium supplementation and risk of event by source and adjudication 


status 












Calcium* 


Placebo* 


Relative risk 




Event source and adjudication 


(N=732) 


(N=739) 


(95% CI) 


p Value 


Myocardial infarction 










Total self-reported events 


31 


14 


O O A IA O r\ 4- A A ~7\ 

2.24 (1 .20 to 4.17) 


0.0099 


Verified self-reported eventsf 


21 


10 


2.12 (1 .01 to 4.47) 


0.047 


Total hospital discharge coding 


24 


17 


A A O /O "7 "7 i_ O AO\ 

1 .43 (0.77 to 2.63) 


0.27 


Total hospital discharge coding and death 


0~7 


To 


A r-A /r\ QA O "70\ 

\ .CM (U.o4 TO iL.( 6) 


U.I o 


certificates 










Verified self-reports/hospital discharge coding and 


o\ 


OH 


■i /in /n qc o c"7\ 
1 .4y (U.oo TO d.Df) 


U.I D 


death certificates 










otroKe 










i oiai seiT-reponeu evenis 


/in 

4U 


OQ 
£LO 


1 .44 (U.yU TO tL.o \ ) 


U.I 4 


veriTieu sen-reponeu evenis-f 


Q1 

o\ 


tLd. 


\ .4*1 (U.OO TO *l.4o; 




Total hospital discharge coding 


26 


20 


1 .31 (0.74 to 2.32) 


0.37 


Total hospital discharge coding and death 


27 


21 


1 ^0 fO 74 to P 97\ 


VJ.OO 


certificates 










Verified self-reports/hospital discharge coding and 


34 


25 


1 .45 (0.88 to 2.49) 


0.15 


death certificates 










*Data are number of women experiencing an incident event. 










flncludes one incident myocardial infarction identified during the verification of other events. 






^Includes six incident strokes identified during the verification of transient ischaemic attacks. 







to having experienced a significant adverse event. 
Despite the differences between self-reports, hospital dis- 
charge codes and verified events, neither the source of 
the event nor the level of adjudication substantially 
altered the relationship between treatment allocation 
and occurrence of either MI or stroke. 

In most cases, the final diagnosis for the non-verified, 
self-reported Mis was related to a disorder of the heart, 
suggesting that miscommunication or misunderstanding 
between the participant and their physician led to the 
error. It has been suggested that the gastrointestinal side 
effects of calcium supplements might be misclassified as 
MI, 7 but we found no evidence that this was the case. In at 
least 43% of non-verified cases, MI was specifically consid- 
ered as a diagnosis and excluded. The most common final 
diagnosis for non-verified, self-reported strokes was TIA, 
suggesting that participants did not understand the distinc- 
tion between these two conditions. This is not surprising 
as TIA is often described as a 'mini-stroke'. In approxi- 
mately one-third of non-verified, self-reported events 
(both MI and stroke) , there were no details of any poten- 
tially related event in the medical records. All these events 
occurred in people who had experienced a previous veri- 
fied event, suggesting that they reported the same event 
more than once, or they reported symptoms that were 
similar to their primary event for which they did not seek 
medical attention. 

There are limited published data that specifically 
address differences between self-reported events and the 
medical records. A systematic review identified 15 studies 
that assessed the accuracy of questionnaires of medical 
history compared with the medical records. 1 The propor- 
tion of illnesses reported in questionnaires ranged from 



30% to 53% of those listed in the medical record. 
Conversely, the medical record listed 36-70% of illnesses 
reported in questionnaires. Reporting of surgical proce- 
dures or hospitalisation in questionnaires appeared to be 
more accurate than reporting of non-surgical illness. The 
findings of this systematic review have been replicated in 
more recent studies. 2 8-10 Further studies have addressed 
the accuracy of self-reported cardiovascular events. For MI 
and stroke, approximately 70-80% of self-reported events 
were confirmed by hospital record review, 2 8-13 which the 
authors suggested is sufficiently accurate for use in 
research studies. 8 9 11 For subtyping of stroke, review of 
hospital records adds useful data to self-reports and dis- 
charge coding. 14 

A comprehensive analysis of reporting of cardiovascu- 
lar events that occurred during the Women's Health 
Initiative clinical and observational studies has been 
published. 10 In these studies, local physician adjudica- 
tors reviewed self-reported data and patient medical 
records for all self-reported cardiovascular events. The 
local adjudicators were able to verify 68% of self- 
reported Mis and 72% of strokes. The local adjudicators 
could only verify MI in 78% of events with a discharge 
code for MI, and stroke in 81% of events with a dis- 
charge code for stroke, suggesting coding data from hos- 
pital admissions may not be accurate, a finding 
consistent with the results of our study. Finally, there was 
no complete agreement between local and central adju- 
dicators. Eighty-one per cent of locally adjudicated Mis 
were verified by central adjudicators, and conversely 
86% of centrally adjudicated Mis were classified as MI by 
local adjudicators. The issue of unreported events was 
not addressed. 
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Our findings are broadly consistent with this previous 
research, although it is uncertain whether self-reports in 
response to open-ended questions in a clinical trial are 
comparable to responses to questionnaires or medical 
records based on a mixture of open-ended and 
closed-ended questions. The lower accuracy of MI self- 
reports in our study compared with other studies might 
be because the participants in our trial were older, had 
more comorbidities, and had more cognitive impair- 
ment than participants in previous studies. Another con- 
tributing factor might be that although cardiovascular 
events were prespecified secondary endpoints, many of 
the participants might not have been aware of this, and 
may have placed less value on reporting these events 
than fractures, which were the primary endpoints of the 
study. In the UK, accuracy of diagnostic coding has 
improved substantially in recent years, 15 although no 
similar data are available for New Zealand. It is possible 
that the magnitude of differences between adjudicated 
events and hospital discharge coding might have 
changed since our study was undertaken. 

Inaccurate self-reports or underreporting of events 
would not be expected to affect the results of a study 
unless the study treatment introduced a systematic bias 
that produced differential rates of inaccurate self- 
reports or underreporting of events. However, when 
the event numbers are small, as in many clinical trials, 
chance differences can substantially impact the study 
results. In our study, the adjudication process led to 
marked changes in the degree of statistical signifi- 
cance, even though the relative risks of MI with 
calcium supplements remained elevated at each stage. 
The relationship between calcium supplements and 
occurrence of MI or stroke was similar regardless of 
whether the events were based solely on hospital dis- 
charge coding or self-reports or final verified events, a 
finding that is supported by the similar increase in vas- 
cular risk associated with calcium supplements across 
randomised trials that used various means of event 
identification. 16 17 

A common approach in clinical trials is to adjudicate 
all significant adverse events centrally, although the 
need to do this remains uncertain. 18 The advantages of 
central adjudication include systematically applying the 
definition of an event, reducing the possibility of differ- 
ential misclassification of events, giving greater confi- 
dence in the validity of the study results, and by 
including suitable triggers, potentially identifying events 
that are missed by local investigators. 18 However, central 
adjudication is unlikely to identify events that are not 
reported to the local investigators, adds significant cost 
and complexity to trials, and has not been shown to 
improve the ability to determine treatment effects in car- 
diovascular trials. 18 

In summary, our results suggest that when the accur- 
acy of an event in clinical trials is critical, even for a rela- 
tively common and serious medical event such as MI or 
stroke, self-reports cannot be relied upon and should be 



independently verified from the medical records. 
However, even if independent verification occurs, our 
results also suggest that a substantial number of events 
will not have been reported. For adverse events of par- 
ticular interest, additional steps, such as searches of hos- 
pital discharge databases, should be considered to 
attempt to identify unreported events. Relying solely 
on non-adjudicated hospital discharge coding will 
lead to similar inaccuracies as using self-reported 
data because some events will be missed and some 
events will be included that would not be verified by 
adjudication. 
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